Reasoning about Form and Content of Multimedia Objects

Due to the pervasive role of multimedia documents (MDs) in nowadays information systems, a vast amount of research has been carried out in the last few years on methods for effectively retrieving such documents from large repositories. This research is still in its infancy, due to the inherent difficulty of indexing documents pertaining to media other than text in a way that reflects their information content and, as a consequence, that significantly impacts on retrieval. Nonetheless, a number of theoretical results concerning sub-problems (e.g. the image retrieval problem) have been obtained and experimented with, and on top of these a first generation of retrieval systems have been built [10] and, in some cases, even turned into commercial products [2, 8]. The distinguishing feature of these multimedia retrieval systems (MRSs), and of the related research models, is the lack of a proper representation and use of the content of non-textual documents: only features pertaining to their form, being most amenable to automatic extraction through digital signal processing (DSP) techniques, are used upon retrieval. But this is disturbing, as documents, irrespective of the representation medium they employ, are to be regarded as information carriers, and as such are to be studied along two parallel dimensions, that of form (or syntax, or symbol) and that of content (or semantics, or meaning). Here, “form” is just a collective name for all those (medium-dependent) features of an information carrier that pertain to the representation and to the representation medium, while “content” is likewise a collective name for those (medium-independent) features that pertain to the slice of the real world being represented, which exists independently of the existence of a representation referring to it. The main thrust of this paper is that a data model for the retrieval of MDs (which we here take as consisting of multiple sub-documents each pertaining to possibly different media, rather than as just non-textual “atomic” documents) not only needs both dimensions to be taken into account, but also requires that each of them be tackled by means of the tools most appropriate to it, and that these sets of tools be integrated in a principled way in order to ensure transparent user access. Concerning the issue of tool appropriateness, we think that, inasmuch as the techniques from DSP (used e.g. in image and audio retrieval) are inadequate to reason about content, those from the field of knowledge representation are inadequate to deal with document form. This study addresses the problem of injecting semantics into MD retrieval by presenting a data model for MDs where sub-documents may be either texts or images. The way this model enforces the interaction between these two media is illustrative of how other media might also be accounted for. Texts and images are represented at the content level as sets of properties of the real-world objects being represented; at this level, the representation is medium-independent, and a unique language for content representation is thus adopted. This data model is logic-based, in the sense that this latter language is based on a description logic (DL – see e.g. [3]). Texts and images are also represented at the form level, as sets of physical features of the objects representing a slice of the world; at this level, the representation is medium-dependent, so